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About  The  National  Science  And  Technology  Council 

The  National  Science  and  Technology  Council  (NSTC)  was  established  by  Executive  Order  on  November  23, 
1993.  This  cabinet-level  council  is  the  principal  means  by  which  the  President  coordinates  science,  space,  and 
technology  policies  across  the  Federal  government.  NSTC  coordinates  diverse  paths  of  the  Federal  research 
and  development  enterprise. 

An  important  objective  of  the  NSTC  is  the  establishment  of  the  clear  national  goals  for  Federal  service  and 
technology  investments  in  areas  ranging  from  information  technologies  and  health  research  to  improving 
transportation  systems  and  strengthening  fundamental  research.  The  Council  prepares  research  and 
development  strategies  that  are  coordinated  across  the  Federal  agencies  to  form  a  comprehensive  investments 
package  aimed  at  accomplishing  multiple  national  goals. 

For  more  information  visit  http://www.ostp.gov/nstc/html/NSTC  Flome.html/. 

About  The  Office  Of  Science  And  Technology  Policy 

The  Office  of  Science  and  Technology  Policy  (OSTP)  was  established  by  the  National  Science  and  Technology 
Policy,  Organization  and  Priorities  Act  of  1976.  OSTP's  responsibilities  include  advising  the  President  in  policy 
formulation  and  budget  development  on  all  questions  concerning  science  and  technology  (S&T);  articulating 
the  President's  S&T  policies  and  programs;  and  fostering  strong  partnerships  among  Federal,  state  and  local 
governments  and  the  scientific  communities  in  industry  and  academe. 

Every  fiscal  year,  OSTP  and  the  Office  of  Management  and  Budget  (OMB)  issue  a  memorandum  entitled 
"Administration  Research  and  Development  Budget  Priorities."  The  memorandum  highlights  the 
Administration's  research  and  development  priorities  and  emphasizes  improving  management  and 
performance  to  maintain  excellence  and  leadership  in  science  and  technology.  The  FY  2008  memorandum  is 
available  at  http://www.ostp.gov/html/budget/2008/m06-17.pdf. 

For  more  information  visit  http://www.ostp.gov  . 
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EXECUTIVE  OFFICE  OF  THE  PRESIDENT 

NATIONAL  SCIENCE  AND  TECHNO  LOOT  COUNCIL 

WASHINGTON.  D  C  JfiWJ 


November  13,  ZUUB 


Dear  Cfrfleaguc. 

3  am  pleased  ki  fLirwaid  this  ducurntnit,  'The  Science  of  Science  Policy:  A  Federal  Reisearch 
Riudmap."  It  was  developed  in  response  to  my  challenge  tor  a  ww  lienee  of  TCfetJCC  polity" 
that  will  begin  lo  address  l he  need  for  bctler  scientific  theories  and  undlyEjenl  tools  for  improving 
our  understanding  of  the  efficacy  and  impact  of  science  and  technology  policy  decisions.  Ii  was 
prepared  hy  ;m  Interagency  Tusk  Group  1 1TG|  commissioned  by  the  National  Science  and 
Technology  Council's  ( NSTC  i  Subcommittee  on  Social.  Behavioral  and  Economic  Sciences 


(SBE). 


This  Roadmap  represents  the  first  organized  description  of  the  emergent  field  of  the  Science  of 
Science  Policy,  outlining  scientific  theories  and  defining  terms  that  encompass  efforts  in  the  Odd 
Them  IVu,  It  highlights  the  potential  for  great Jy  increasing  the  knowledge  hose  and  providing 
needed  insights  to  improve  the  data,  loots  and  methods  thal  would  enable  a  mure  rigorous  arid 
quantitative  bub  for  science  and  technology  policy.  The  Roadmap  identifies  ten  major  science 
qucsiions  grouped  into  three  broad  rhemes:  Undemanding  Science  and  Innovation;  Investing  in 
Science  and  Innovation:  and  Using  the  Science  of  Science  Policy  to  Address  National  Priorities. 

Agencies  and  departments  across  the  Federal  Government  face  similar  challenges  when  setting 
scientific  priorities  and  assessing  the  effectiveness  of  current  and  planned  investments.  By 
working  tugcfilker  lo  address  t3ie.se  themes  and  questions,  share  best  practice1;  and  culluhtiraic  on 
fundamental  principles,  we  will  greatly  enhance  our  ability  lorinxlmfec  our  critical  investments 
in  science  and  technology. 


Sincerely, 


John  H.  Marburgcr,  UJ  \ A 

Director,  Office  of  Science  and  Technology  Policy 
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Executive  Summary 


What  is  the  Science  of  Science  Policy? 


The  science  of  science  policy  (SoSP)  is  an  emerging  field  of  interdisciplinary  research,  the  goal  of  which  is  to 
provide  a  scientifically  rigorous,  quantitative  basis  from  which  policy  makers  and  researchers  can  assess  the 
impacts  of  the  Nation's  scientific  and  engineering  enterprise,  improve  their  understanding  of  its  dynamics, 
and  assess  the  likely  outcomes.  Research  in  SoSP  could  be  utilized  by  the  Federal  Government,  and  the  wider 
society  in  general,  to  make  better  R&D  management  decisions. 


The  term  "science  of  science  policy"  was  first  used  by  Dr.  John  H. 
Marburger  III,  the  President's  Science  Advisor  and  Director  of  the  Office 
of  Science  and  Technology  Policy  (OSTP)  in  his  keynote  address  to  the 
American  Association  for  the  Advancement  of  Science  (AAAS)  Science 
and  Technology  Policy  Forum  in  April,  20051.  This  was  expanded  in 
a  May  2005  Science  magazine  editorial  (see  Appendix  B).  In  that 
editorial.  Dr.  Marburger  called  for  the  creation  of  a  community  of 
practice  that  would  create  the  data  sets,  tools,  and  methodologies 
needed  to  assist  science  policy  decision  makers  as  they  invest  in  Federal 
research  and  development  and  make  science  policy  decisions.  A 
National  Science  and  Technology  Council  (NSTC)  Interagency  Task  Group 
(ITG)  was  created  in  2006  to  develop  a  coordinated  Federal  approach  to 
the  science  of  science  policy  to  meet  these  challenges. 

A  National  Imperative  for  a  Science  of  Science  Policy 

Development  of  the  science  of  science  policy  is  critically  important  to 
our  Nation's  ability  to  benefit  most  effectively  from  R&D  investments. 

In  2007,  the  U.S.  Federal  government  R&D  budget  totaled  $139  billion1. 


Primary  Conclusion 
of  the  Interagency 
Working  Group: 

"Expert  judgment"  remains 
the  best  available  decision 
support  tool  for  science 
policy  makers,  but  a  nascent 
community  of  practice  is 
emerging  in  the  science 
policy  arena  that  holds 
enormous  potential  to  provide 
rigorous  and  quantitative 
decision  support  tools  in  the 
near  future.  Support  and 
development  of  this  emerging 
community  of  practice 
can  provide  the  Federal 
government  with  these  much- 
needed  decision  tools. 


Although  the  importance  of  public  investments  in  science,  technology, 

and  innovation  is  understood,  the  rationale  for  specific  scientific  investment  decisions  lacks  a  strong  theoretical 
and  empirical  basis.  Accordingly,  given  the  magnitude  of  the  Federal  investment  and  the  importance  of  that 
investment  to  our  Nation,  science  policy  decision  makers  must  have  at  their  disposal  the  most  rigorous  tools, 
methods  and  data  that  will  enable  them  to  develop  sound  and  cost-effective  investment  strategies. 


The  ITG  undertook  a  literature  review  to  determine  the  state  of  the  science  to  date.  A  questionnaire  was  also 
circulated  to  Federal  agencies  to  ascertain  what  methods  are  currently  in  use  for  programmatic  investment 
decision  making,  as  well  as  to  ask  what  tools  and  resources  are  needed  by  Federal  agencies  that  are  currently 
unavailable.  The  ITG  found  that: 


•  There  is  a  well  developed  body  of  social  science  knowledge  that  could  be  readily  applied  to  the 
study  of  science  and  innovation. 

•  Although  many  Federal  agencies  have  their  own  communities  of  practice,  the  collection  and 
analysis  of  data  about  the  science  and  scientific  communities  they  support  is  heterogeneous  and 
unsystematic. 

•  Agencies  are  using  very  different  models,  data  and  tools  to  understand  their  investments  in  science 
and  technology. 

•  The  data  infrastructure  is  inadequate  for  decision-making. 
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•  New  tools  and  data  sets  could  be  developed  and  used  to  quantify  the  impact  that  the  scientific 
enterprise  has  had  on  innovation  and  competitiveness. 

This  Federal  Research  Roadmap  identifies  three  broad  theoretical  themes,  with  10  underlying  scientific  ques¬ 
tions  within  those  themes,  and  makes  several  recommendations  on  how  to  address  those  questions.  More 
generally,  the  following  next  steps  are  recommended  for  overall  implementation  of  the  Roadmap. 

Next  Steps 


•  Federal  government  agencies  should  work  in  concert  to  establish  a  theoretical  and  empirical 
framework  to  understand  the  science  and  engineering  enterprise  within  the  context  of  the  science 
of  science  policy.  Tentatively  described  in  this  report  as  a  "Federal  Innovation  Framework,"  analyses 
could  be  performed  on  how  Federal  investments  and  policy  decisions  affect  the  Nation's  system  of 
innovation. 

•  Establish  interagency  research  priorities  to  address  the  scientific  challenges  confronting  the  unique 
science  policy  analysis  needs  of  Federal  science  and  technology  agencies. 

•  Encourage  investment  in  the  development  and  use  of  emerging  tools,  methods,  data,  and  data 
infrastructure  to  enable  science  policy  decision  makers  to  base  investment  decisions  on  more 
rigorous  and  quantitative  analyses. 

•  Hold  a  public,  international  workshop  to  discuss  the  scientific  basis  of  the  Roadmap  and  its 
implementation  in  the  policy  arena.  The  workshop  will  serve  to  further  inform  the  U.S.  Federal 
approach  to  the  science  of  science  policy  in  a  broader,  international  context. 
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Introduction 


The  National  Imperative 

Federal  investments  in  science  and  technology  have  had  an  enormous  impact  on  innovation,  economic  growth, 
and  social  health  and  well-being.  In  addition  to  furthering  these  impacts,  future  investments  by  the  Federal 
government  will  be  critical  in  many  arenas,  such  as  mitigating  the  consequences  of  global  climate  change, 
exploring  new  energy  sources,  defending  against  external  threats,  and  maintaining  international  competitive¬ 
ness.  Given  the  importance  of  those  investments,  it  is  imperative  that  science  policy  decision  makers  have  at 
their  disposal  the  most  rigorous  tools,  methods  and  data  that  will  enable  them  to  develop  sound  investment 
strategies. 

And  yet  the  science  policy  analysis  community  does  not  have  the  best  tools,  methods  and  data  that  would 
allow  decision  makers  to  make  and  manage  such  future  investments  optimally.  As  a  result,  science  policy 
discussions  are  frequently  dominated  by  advocates  for  individual  scientific  fields  who  argue  for  their  particular 
interests,  but  leave  policy  makers  with  little  ability  to  objectively  discriminate  between  investment  options. 
Policy  decisions  may  be  based  upon  past  practices  or  data  trends  that  may  not  always  accurately  reflect  cur¬ 
rent  conditions. 

Across  the  Federal  government,  there  is  an  urgent  need  for  rigorous  analysis  that  can  inform  Federal  research 


Climate  change  is  one  of  the  major  science  challenges  that  would  benefit  from 
science  policy  analysis  and  decision  making. 

Linear  trend  of  annual  temperatures,  National  Climactic  Data  Center. 
National  Oceanic  and  Atmospheric  Administration, 
http://www.ncdc.noaa.gov/oa/climate/globalwarming.htmlffq4. 


and  development  decision-making.  Investments  in  many  strategically  important  frontiers  of  science  and 
engineering  and  the  allocation  of  Federal  resources  across  a  complex  and  decentralized  national  research  and 
development  (R&D)  portfolio  must  be  guided  by  the  best  data  and  analysis  available.  A  "science  of  science 
policy",  first  named  by  the  President's  Science  Advisor,  Dr.  John  Marburger,  must  be  developed. 

The  importance  of  this  challenge  derives  from  the  magnitude  and  centrality  of  the  contribution  that  science 
and  technology  make  to  the  U.S.  economy.  In  2007,  the  U.S.  Federal  government  R&D  budget  totaled  $139  bil- 
lion2,  affirming  the  importance  of  Federal  investments  in  science  and  technology.3  It  is  imperative  to  advance 
the  scientific  basis  of  science  policy  so  that  limited  Federal  resources  are  invested  wisely.  Scientific  models 
must  be  developed,  along  with  methods  of  collecting  real-time  quantitative  and  qualitative  data  so  that  future 
policy  decisions  are  based  on  sound  science  and  informed  by  meaningful  metrics.  Retrospective  analysis  is  also 
needed,  to  analyze  the  impact  of  Federal  investments  on  scientific  discovery  and  innovation,  the  economy, 
and  society.  In  this  way,  past  investments  may  help  inform  future  decisions,  refine  the  accuracy  of  models,  and 
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maintain  the  nation's  dominance  in  the  scientific  arena. 

While  federal  decision-makers  grapple  with  competing 
priorities  to  make  the  best  decisions  for  their  own  agen¬ 
cies,  the  rest  of  the  world  is  vastly  increasing  its  scientific 
investments,  and  there  is  increased  foreign  competition 
for  scientific  ideas  and  talent.  The  United  Kingdom  has 
established  a  Department  for  Innovation,  Universities 
and  Skills;  Saudi  Arabia  has  invested  $6  billion  to  estab¬ 
lish  a  University  of  Science  and  Technology;  and  almost 
every  developed  European  and  Asian  country  is  aggres¬ 
sively  investing  in  and  competing  for  scientific  talent. 

Thus,  science  and  technology  policy  and  the  choice  of 
scientific  investments  remain  at  the  forefront  of  the 
national  debate. 

The  Scientific  Challenge 

Developing  a  science  of  science  policy  will  require  the  implementation  of  a 
rigorous  and  systematic  research  program  that  is  focused  on  addressing  a 
number  of  scientific  and  data  challenges,  ensuring  that  the  needs  of  Federal 
R&D  agencies  and  senior  decision  makers  are  met.  Its  creation  as  a  scientific 
field  of  study  requires  the  development  of  analytical  tools,  data  bases,  and 
management  processes  capable  of  providing  reliable  information  and  the 
best  possible  basis  for  the  allocation  of  public  funds  available  to  support 
Federal  R&D.  It  could  be  built  upon  the  emerging  community  of  science 
in  this  area,  and  existing  expertise  in  policy  analysis  that  is  resident  in  the 
Federal  science  agencies  and  elsewhere  in  the  Federal  government.  A 
community  of  practice  in  academia  and  in  the  private  sector  could  also  be 
developed  that  would  provide  support  to  Federal  policy  makers. 

Current  science  and  technology  investment  decisions  are  based  on  analyses 
that  lack  a  strong  theoretical  and  empirical  basis.  Increasingly,  economic 
value  is  based  on  generating  and  selling  ideas,  rather  than  physical  goods 
and  agricultural  products,  but  the  current  social  scientific  and  statistical 
infrastructure  has  not  kept  pace  with  this  change  in  the  nature  of  economic  activity.  Indeed,  while  it  is  a 
common  belief  that  innovation  is  closely  related  to  investments  in  science  and  technology,  there  is  actually  a 
limited  theoretical  and  scientific  foundation  underlying  such  beliefs. 

But  there  is  an  emerging  view  that  many  facets  of  the  U.S.  innovation  ecosystem  have  become  too  complex 
for  expert  judgment  alone  to  be  an  effective  decision  support  tool.  Science  continues  to  accelerate,  and 
multidisciplinary  collaborations  are  becoming  more  common;  as  a  result,  the  complexity  of  the  scientific 
endeavor  is  surpassing  the  ability  of  experts  within  particular  scientific  disciplines  to  understand  its  totality. 

The  tenuous  nature  of  the  scientific  links  between  investments  in  science  and  desired  outcomes  is  evident 
from  the  variety  of  ways  in  which  outcomes  are  characterized:  sometimes  by  the  entity  funding  or  conducting 
R&D  (e.g.,  universities,  governments,  or  businesses),  sometimes  by  the  phase  of  discovery  (e.g.,  basic  research, 
applied  research,  or  development),  and  sometimes  in  terms  of  the  end  products  (e.g.,  products,  processes, 
organizations,  or  knowledge). 


TuffCell  bi- polar  plates 
Argonne  National 
Laboratory 
http://www.anl.gov/ 


Scope  of  the  Science  of  Science  Policy: 

The  science  of  science  policy  includes  basic 
and  applied  research,  as  well  as  technology 
development,  demonstration,  and  deployment. 
It  includes  operational  science  and  technology, 
from  early  to  later  stages  in  operational 
life  cycles.  It  is  thus  comprehensive  of  the 
entire  spectrum  of  innovation  in  science 
and  technology  activities,  from  high-risk, 
undirected  activities  to  low  risk,  applied 
activities. 
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This  complexity  creates  a  number  of  challenges  for  science  policy,  and  is  the 
subject  of  a  vigorous  scientific  debate.  These  challenges  include  determining 
the  appropriate  roles  of  various  Federal  agencies  in  different  disciplines, 
addressing  a  host  of  agency  policy  and  investment  questions  about  the 
appropriateness  of  disciplinary  or  institutional  portfolios,  and  understanding 
the  relative  value  added  of  different  fiscal  policy  stimuli.  In  order  to  address 
this  need,  the  Office  of  Management  and  Budget  and  the  Office  of  Science  and 
Technology  Policy  have  encouraged  Federal  science  and  technology  agencies  to 
work  cooperatively  to  bring  the  emerging  science  of  science  policy  community 
of  practice  into  a  mature  state,  via  the  annual  "Administration  Research  and 
Development  Budget  Priorities"  memoranda. 45 


The  Interagency  Task  Group  ( ITG )  Charge  and  Findings 

In  recognition  of  the  importance  of  developing  a  science  of  science  policy,  NSTC  established  the  Science  of 
Science  Policy  ITG,  under  the  purview  of  the  Subcommittee  on  Social,  Behavioral,  and  Economic  Sciences  (SBE). 
The  Co-Chairs  of  the  ITG  are  representatives  of  the  U.S.  Department  of  Energy  (DOE)  and  the  National  Science 
Foundation  (NSF).  Other  participating  agencies  include  the  Departments  of  Agriculture  (USDA),  Commerce 
(DOC),  Defense  (DOD),  Education  (DOEd),  Health  and  Fluman  Services  (DHFIS),  Flomeland  Security  (DEIS), 
Interior  (DOI),  State,  Transportation  (DOT),  and  Veterans  Affairs  (VA),  as  well  as  the  National  Aeronautics  & 
Space  Administration  (NASA),  the  National  Institutes  of  Health  (N I H),  the  US  Geological  Survey  (USGS),  the 
Centers  for  Disease  Control  and  Prevention  (CDC),  the  National  Institute  of  Standards  and  Technology  (NIST), 
the  Environmental  Protection  Agency  (EPA),  the  National  Oceanic  and  Atmospheric  Administration  (NOAA), 
OSTP,  and  OMB.  The  Interagency  Task  Group  had  a  number  of  charges  as  articulated  in  its  Charter  (see 
Appendix  A). 

First,  the  ITG  developed  the  following  definition  of  the  science  of  science  policy: 

"The  science  of  science  policy  is  an  emerging  interdisciplinary  research  area  that  seeks  to  de¬ 
velop  theoretical  and  empirical  models  of  the  scientific  enterprise.  This  scientific  basis  can  be 
used  to  help  government,  and  society  in  general,  make  better  R&D  management  decisions  by 
establishing  a  scientifically  rigorous,  quantitative  basis  from  which  policy  makers  and  research¬ 
ers  may  assess  the  impacts  of  the  Nation's  scientific  and  engineering  enterprise,  improve  their 
understanding  of  its  dynamics,  and  assess  the  likely  outcomes.  Examples  of  research  in  the 
science  of  science  policy  include  models  to  understand  the  production  of  science,  qualitative 
and  quantitative  methods  to  estimate  the  impact  of  science,  and  processes  for  choosing  from 
alternative  science  portfolios." 

Next,  the  ITG  completed  four  tasks  to  address  its  charge:  1)  A  review  of  current  Federal  efforts  related  to 
the  science  of  science  policy;  2)  Examination  of  the  data  that  are  available  for  analysis;  3)  Development 
of  a  literature  synthesis  that  brings  together  academic  research  from  many  different  disciplines;  and  4) 
Development  of  a  roadmap  that  would  chart  a  path  forward  for  the  Federal  government  to  build  a  community 
of  practice  as  well  as  tools  in  the  science  of  science  policy.  This  work  was  assisted  by  groundwork  laid  by  the 
new  grant  program  in  NSF's  Directorate  for  Social,  Behavioral,  and  Economic  Sciences:  the  Science  of  Science 
and  Innovation  Policy  (SciSIP).  The  program,  although  independent  of  the  ITG,  provided  many  insights, 
particularly  with  regard  to  the  literature  synthesis. 


Graduate  student 
Lawrence  Berkeley  National 
Laboratory 
http://www.lbl.aov/. 
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The  first  task,  a  review  of  current  Federal  agency  efforts  that  could  be  viewed  as  supportive  of  the  science  of 
science  policy,  found  the  following: 

•  A  number  of  Federal  agencies  have  developed  substantial  capabilities  for  R&D  program  planning 
and  evaluation  to  develop  and  manage  their  R&D  portfolios,  measure  program  progress  toward 
strategic  goals,  and  to  address  other  operational  and  policy  issues.  The  vast  majority  of  those 
capabilities  relies  upon  "expert  judgment"  and  does  not  use  quantitative  models  or  decision 
analysis  tools.  This  is  consistent  with  findings  from  several  reports  done  by  NAS  in  the  1990s  that 
identified  expert  judgment  as  the  primary  tool  available  to  science  policy  makers.6  The  NAS  went 
further  in  a  2008  report  which  stated  that  "the  most  effective  mechanism  for  evaluating  investment 
efficiency  of  R&D  programs  is  an  expert-review  panel."  7  It  may  be  noted  that  these  reports  are 
largely  silent  on  evaluations  across  programs  or  the  establishment  of  investment  priorities  among 
programs  in  different  fields. 
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The  NAS  has  published  four  reports  since  1993  that  examine  how  to  assess  the  benefits 
and  effectiveness  of  Federal  investment  in  science  and  technology. 


•  Science  and  technology  agencies  expended  considerable  effort  in  response  to  the  requirements  for 
strategic  planning  and  evaluation  imposed  by  the  1993  Government  Performance  and  Results  Act 
(GPRA).  This  attention  to  management,  planning,  and  evaluation  was  amplified  in  2001,  as  science 
and  technology  programs  increasingly  became  the  subject  of  OMB's  Program  Assessment  Rating 
Tool  (PART)  assessments.  During  this  period,  many  agencies  found  creative  ways  to  utilize  expert 
judgment  and  other  types  of  quantitative  and  qualitative  decision  analysis  tools  to  meet  GPRA  and 
PART  requirements. 

The  ITG  next  examined  data  needs  and  found  the  following: 

•  The  agencies  responsible  for  collecting  national  science  and 
economic  statistics,  such  as  NSF  and  DOC,  are  increasingly 
focused  on  better  ways  to  gauge  the  effects  of  R&D 
investment  on  the  production  of  new  knowledge  and 
technology,  on  innovation,  and  their  ripple  effects  throughout 
the  economy. 

•  Dr.  Marburger  addressed  the  Organization  for  Economic 
Cooperation  and  Development  (OECD)  regarding  Science 
of  Science  Policy  needs  in  July  2006. 8  Since  that  time,  the 
statistical  and  analytic  committees  of  the  OECD  -  to  which  the 
United  States  has  been  a  major  contributor  and  participant 
-  have  devoted  much  effort  to  measuring  the  strength  and 


"Complexity  economics 
is  part  of  the  larger  area 
of  complex  adaptive 
systems  that  incorporates 
methods  from  the  study  of 
such  systems  in  physics, 
biology,  computer  science, 
and  other  fields." 

-  Science  of  Science  Policy:  An 
Exploration  of  Literature  and 
Practice,  created  on  behalf 
of  the  Science  of  Science 
Policy  ITG  by  the  Science  and 
Technology  Policy  Institute 
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effectiveness  of  the  innovation  infrastructures  of  member  countries  and  their  contributions  to 
national  economic  growth. 

•  The  research  data  infrastructure  suited  to  examining  science  and  innovation  policy  is  insufficient  to 
support  today's  requirements.  Resources  are  wasted  because  researchers  are  often  forced  to  use 
data  that  is  not  collected  for  research  purposes  or  within  the  appropriate  scientific  context.  As  a 
result,  their  analytical  work  frequently  cannot  be  generalized.  Limited  access  to  data  often  means 
that  analytical  work  cannot  be  replicated. 


Nobel  Laureate 
Daniel 
Kahneman: 

The  2002  Nobel 
Prize  was  awarded 
to  psychologist 
Daniel  Kahneman 
for  demonstrating 
that  in  situations 
with  uncertainty, 
human  judgment  often  exploits 
rules  of  thumb  which  systematically 
contradict  fundamental  propositions  in 
probability  theory. 

An  important  finding  in  the  science 
policy  context  is  that  individuals  are 
much  more  sensitive  to  the  way  an 
outcome  deviates  from  a  reference 
level  (often  the  status  quo)  than  to 
the  absolute  outcome.  When  faced 
with  a  sequence  of  decisions  under 
risk,  individuals  thus  appear  to 
base  each  decision  on  its  gains  and 
losses  in  isolation  rather  than  on  the 
consequences  of  a  decision  for  their 
wealth  as  a  whole. 

2002  Nobel  Prize  Press  Release 


Finally,  the  ITG  commissioned  a  literature  synthesis  designed  to 
review  academic  research  in  science  policy  and  related  fields. 

This  synthesis  examined  economic  theory,  social  and  behavioral 
sciences,  the  physical  and  biological  sciences,  and  a  host  of  other 
disciplines  that  could  provide  insights  into  analytical  methods, 
tools,  and  data  sets  useful  to  the  science  policy  community, 
such  as  the  emerging  theories  of  complexity  in  mathematics  and 
systems  biology.  The  ITG  also  reviewed  other  work,  most  notably 
a  report  by  the  NAS's  Committee  on  Science,  Engineering  and 
Public  Policy  (COSEPUP)9  and  ongoing  work  by  the  OECD.10  This 
synthesis  led  to  the  conclusion  that  systems  level  analysis  from 
biology,  complexity  theory,  social  network  analysis,  industrial 
dynamics  and  other  disparate  fields  have  rich  potential  for  the 
future  of  the  science  of  science  policy.  A  primary  conclusion  of 
the  ITG  is  that  practitioners  from  these  related  fields  should  begin 
working  to  mine  that  potential  for  SoSP. 

Based  upon  these  three  undertakings  and  their  subsequent 
findings,  the  ITG  formed  the  primary  scientific  themes  and 
questions  which  comprise  the  current  state  of  the  Science  of 
Science  Policy.  These  were  incorporated  into  the  Roadmap,  which 
sets  forth  a  series  of  recommendations  to  advance  the  field  of 
SoSP. 
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The  ITG  identified  three  scientific  themes  and  10  major  questions  faced  by  scientific  agencies  if  they  are  to  help 
support  the  development  of  the  SoSP  community.  They  are  listed  under  broad  conceptual  categories  below; 
the  associated  recommendations  are  identified  in  the  next  section. 

Theme  1:  Understanding  Science  and  Innovation 

Establishing  a  scientific  framework  for  understanding 
science  and  innovation  is  fundamental  if  policy 
makers  are  to  understand  the  ways  in  which  their 
decisions  are  likely  to  play  out.  Because  innovation 
in  scientific  disciplines  relies  on  human  achievement, 
interaction  and  behavior,  research  should  be  grounded 
in  the  core  social  and  behavioral  science  disciplines 
of  economics,  sociology,  psychology,  and  political 
science.  Theoretical  models  of  innovation  must  be 
developed,  in  the  same  way  that  economists  have 
developed  theoretical  models  of  economic  activity. 

This  will  provide  a  context  for  data  collection.  Just 
as  the  Federal  Reserve  Board's  econometric  model  is 
based  on  a  strong  theoretical  foundation  describing 
economic  behavior,  a  theoretical  model  could  be  used 
to  develop  the  empirical  framework  upon  which  to 
base  investment  decisions  in  science. 

The  following  list  identifies  the  key  questions  that  must  be  addressed  in  order  to  develop  such  a  scientific 
framework. 

Question  1:  What  Are  The  Behavioral  Foundations  Of  Innovation? 

Innovation,  the  act  of  creating  or  inventing  new  ideas 
or  methods,  has  a  basis  in  social  and  behavioral  activity. 
The  examination  of  the  foundations  of  our  social  and 
economic  systems  by  social  scientists  has  led  to  a 
deeper  understanding  of  economic,  social,  and  cognitive 
principles,  resulting  in  a  number  of  Nobel  prizes,  yet 
there  has  been  little  focus  on  the  social  and  behavioral 
foundations  of  innovation  in  science  and  technology.  Initial 
findings  may  provide  a  basis  for  the  development  of  more 
dynamic  research  programs  in  this  area.  For  example, 
psychologists  funded  by  NSF  have  developed  models 
of  cognitive  processes  across  individuals  and  groups 
intended  to  promote  innovation.  Psychologists  have  also 
studied  trans-disciplinary  research  teams.  Examples  of 
these  include  partnership  teams  between  biomedical  and 
nanotechnology  industries,  the  collaboration  between 
academic  and  non-academic  scientists  (in  fields  such  as 
hydrology,  soil  and  water  science),  and  the  development 


Nobel  Laureate 
Ronald  Coase: 

The  1991  Nobel  Prize  was 
awarded  to  economist 
Ronald  Coase  for  finding 
that  the  institutional 
structure  of  the  economy 
may  be  explained  by 
the  relative  costs  of 
different  institutional 
arrangements,  combined 
with  the  parties'  efforts  to 
keep  total  costs  at  a  minimum. 

Alongside  price  formation,  the  formation 
of  the  institutional  structure  is  regarded  as 
an  integral  step  in  the  process  of  resource 
distribution. 

1991  Nobel  Prize  Press  Release 


Figure  1.  The  Target  for  the  Federal  Funds  Rate 


The  Federal  Reserve  uses  its  econometric  model  to 
set  monetary  policy. 

http://www.econedlink.org/lessons/index. 

cfm?  lesson  =  EM712&paae=teacher 
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of  virtual  social  networks  (Internet  based).  Economists  and  sociologists  have  begun  to  extend  models  of  the 
impact  of  different  incentive  and  organizational  structures  on  human  behavior  to  the  scientific  enterprise. 

Some  Federal  agencies,  such  as  NIH  and  DOE,  have  begun  to  use  social  network  analysis  techniques  to 
understand  the  process  of  innovation. 

Finding:  The  ITG's  review  of  the  literature  suggests  that  while  there  is  a  well-developed  understanding 
of  human  and  social  behavior  in  multiple  disciplines  such  as  economics,  psychology,  and  sociology,  this 
understanding  has  yet  to  be  applied  to  the  study  of  innovation  within  the  scientific  enterprise,  leaving 
enormous  gaps  in  scientific  knowledge.  For  example:  how  does  the  discovery  process  work  at  the  individual 
and  team  level?  Flow  could  creative  insights  be  stimulated?  Which  institutional  structures  facilitate  the 
discovery  to  innovation  cycle? 

Question  2:  What  Explains  Technology  Development.  Adoption  And  Diffusion? 

Technology  is  the  application  of  science,  especially  to  commercial  and  industrial  objectives.  Of  key  interest, 
then,  is  advancing  understanding  of  how  technology  is  adopted,  who  adopts  technology  and  how  it  spreads 
through  society.  The  current  plethora  of  theories  and  models  that  could  inform  Federal  decision-makers  in 
moving  R&D  to  deployment,  operations,  and  private  sector  applications,  needs  to  be  advanced  and  applied 
to  the  current  context.  This  would  help  inform  decisions  as  to  whether  funding  increases,  tax  changes, 
regulatory  changes,  or  other  policy  interventions  are  more  or  less  likely  to  stimulate  technology  development, 
adoption  and  diffusion. 

It  is  clear  that  technology  adoption  and  diffusion  is  a  complex  process;  there  is  not  a  linear  succession  from 
basic  research  to  the  market,  via  applied  research,  technical  development,  production  and  marketing. 
Businesses  and  government  rely  upon  various  decision-making  tools  that  promote  technology  adoption  - 
including  considerations  about  the  return  on  investment  (ROI),  intellectual  property  valuation,  consumer 
preferences,  and  many  other  factors  that  enable  technology  adoption  -  but  how  those  decision  rules  interact 
and  lead  from  the  maturation  of  a  scientific  concept  into  a  market-based  technology  is  currently  unclear.  It  is 
also  important  to  understand  how  leaders  (those  likely  to  take  the  greatest  risk  and  adopt  a  technology  early) 
behave  within  a  specific  social  network.  In  some  instances,  less  scientific  considerations  such  as  the  reputation 
of  the  early  adopter  play  a  key  role  in  whether  a  technology  gains  access  and  diffuses  successfully. 

There  are  indications  that  answers  can  be  found.  The  role  of  economic  incentives  in  technology  adoption 
has  been  clear  since  Griliches's  analysis  of  the  adoption  of  hybrid  corn  in  developing  countries.11  Fluman 
capital  is  also  a  critical  element:  researchers  have  found  that  technology  diffuses  through  social  networks, 
since  these  reduce  learning  costs,  enhance  usability  and  sustainability,  and  create  a  social  incentive  structure. 
Researchers  at  NIST,  for  example,  have  learned  that  collecting  data  on  businesses  and  collaborating  with 
academic  researchers  enables  government  to  do  the  analyses  that  promote  economic  growth  by  encouraging 
the  development  of  "disruptive  innovations".  Depending  entirely  on  high-tech  startups  to  develop  disruptive 
innovations  and  introduce  them  to  market— waiting  for  small  technology  firms  to  mature  into,  or  merge  with, 
larger  firms,  thereby  transforming  industries  from  the  bottom  up— is  a  strategy  that  is  both  slow  and  uncertain, 
but  NIST  has  learned  that  technology  adoption  can  be  accelerated  through  certain  targeted  investments. 
Current  research  also  suggests  that  large  firms  increasingly  have  a  need  for  external  partners  to  help  them 
overcome  internal,  as  well  as  external,  barriers  to  the  development  of  disruptive  innovations.12 
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Finding:  The  investigation  of  technology  adoption  and  diffusion  has  been  largely  confined  to  the  academic 
realm.  Some  government  agencies,  such  as  NIST,  have  collected  data  in  a  scientific  fashion,  and  have 
supported  basic  research.  However,  challenges  remain,  such  as  the  development  of  technology  adoption 
models,  as  well  as  research  on  full  systems  approaches  to  mapping  science,  technology,  and  innovation.  This 
research  could  be  significantly  advanced  by  developing  stronger  links  between  the  academic  and  practitioner 
communities. 


DEFINITION:  DISRUPTIVE  INNOVATION: 

The  term  "disruptive  technology"  was  first  described  by  Christensen  as  the  creation 
of  cheaper,  simpler-  to- use  versions  of  existing  products  (in  contrast  to  "sustaining 
technologies",  which  provide  incremental  improvements  on  existing  products).  Now  the 
term  "disruptive  innovation"  is  commonly  used  to  describe  any  technological  innovation, 
product,  or  service  that  uses  a  "disruptive"  strategy,  rather  than  a  "sustaining"  strategy, 
to  impact  existing  dominant  technologies  or  status  quo  products  in  a  market.  Disruptive 
innovations  may  create  entirely  new  markets,  or  new  customers  within  existing 
markets. 

Clayton  Christensen,  The  Innovator's  Dilemma  (Harvard  Business  School  Press),  1997. 


Question  3:  How  And  Why  Do  Communities  Of  Science  And  Innovation  Form  And  Evolve? 


Communities  of  science  provide  the  backdrop  for 
promoting  scientific  discovery  and  innovation. 

Building  on  the  existing  understanding  of  how  such 
communities  evolve  would  have  clear  implications  for 
investment  decisions.  Research  funding,  for  example, 
could  be  structured  to  encourage  the  formation  of 
new  communities,  as  is  currently  occurring  through 
the  large  Federal  investment  in  the  nanoscience 
and  synthetic  biology  communities.  Studying  the 
behavior  and  formation  of  communities  could 
avoid  unnecessary  duplication,  since  a  feature  of 
communities  is  the  joint  study,  documentation,  and 
communication  of  scientific  advances.  Finally,  using 
communities  as  the  unit  of  analysis  is  another  way 
of  tracking  the  scientific  impact  of  investments.  It  is 
apparent  that  the  increasing  complexity  of  science 
means  that  institutional  and  disciplinary  boundaries 
are  no  longer  the  organizing  principle  for  the 
development  of  communities.  Social  scientists  have 
been  able  to  use  new  tools  to  advance  understanding 
of  social  network  theory,  although  the  data 
requirements  are  complex  and  evolving.  The  field  of 
study  of  virtual  organizations  is  also  emerging,  with  a 
first  solicitation  by  NSF's  Office  of  Cyberinfrastructure. 

A  number  of  research  agencies  have  developed  the 
infrastructure  to  study  communities.  For  example, 
USDA  science  agencies  are  using  its  Current  Research 
Information  System  (CRIS)  to  keep  track  of  performers. 


SCI  ENCE  POLI CY  I  N  ACTI  ON: 

Bl  OTECHNOLOGY  Rl  SK  ASSESSMENT 
RESEARCH 

USDA  is  funding  work  on  the  multi¬ 
dimensional  issues  regarding  the  adoption 
and  diffusion  of  biotechnology.  These  issues 
revolve  around  agriculture  and  food  systems, 
markets  and  consumers,  businesses, 
institutions,  and  social  issues. 


Ag  Engineer  and  Corn,  Agricultural 
Research  Service,  U.S.  Department 
of  Agriculture 

The  changing  agriculture  enterprise  in  the 
21st  Century  includes  increasing  use  of 
transgenic  crops.  While  these  crops  have 
been  widely  adopted  in  the  United  States 
to  great  benefit,  there  remain  a  number 
of  concerns  about  their  adoption  that  have 
created  barriers  to  their  sale  and  export  to  a 
number  of  countries. 
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"Emerging  economies  learned 
a  key  lesson:  investment  in 
innovation  capacity  is  the  key  to 
higher  productivity,  higher  wages 
and  higher  economic  growth. 
Emerging  economies  are  investing 
in  research  and  virtual,  physical 
and  educational  infrastructure. 
Global  companies  are  establishing 
additional  innovation  capabilities 
in  the  emerging  world  as  they 
increasingly  collocate  R&D  with 
new  market  opportunities.  While 
the  United  States  is  the  world’s 
strongest  innovator  nation 
today,  a  wide  range  of  surveys 
shows  that  many  companies 
plan  to  establish  high  value  and 
knowledge-intensive  operations 
offshore,  including  R&D,  and 
that  emerging  economies  are 
now  among  the  most  attractive 
destinations  for  that  investment. 

Five  for  the  Future.  Council  on 
Competitiveness,  2007. 


Relational  databases  are  being  developed  by  academics,  and 
NSF  has  funded  the  study  of  complex  distributed  project  teams 
developed  under  individual  solicitations,  such  as  its  Information 
Technology  Research  program.  DOE  has  funded  three  separate  social 
network  studies  targeted  at  the  high-performance  computing  and 
nanoscience  communities. 

Finding:  Although  each  Federal  agency  has  its  own  community  of 
practice,  the  collection  and  analysis  of  data  about  the  scientists 
and  the  communities  supported  by  those  Federal  agencies  is 
heterogeneous  and  unsystematic.  There  is  little  analysis  of  the  way 
in  which  the  practice  of  science  has  become  distributed  across  space, 
time,  and  research  areas  as  a  result  of  computational  advances.  As 
a  result,  there  is  little  understanding  of  how  scientific  communities 
respond  to  changes  in  funding  within  research  areas  and  across 
national  boundaries,  or  to  changes  in  program  foci.  For  example, 
how  important  are  national  and  international  human  capital  flows? 
What  is  the  role  of  the  Internet  and  cybertools  in  communicating 
scientific  ideas  within  and  across  communities?  Flow  do  different 
mixes  of  research  performers  (industry.  Federal  laboratories  and 
universities)  influence  the  development  of  science  communities? 


Theme  1  Recommendations 


The  ITG  recommends  that  Federal  government  agencies  work  in  concert  to  establish  a  theoretical  and 
empirical  framework  to  understand  the  science  and  engineering  enterprise  within  the  context  of  the  Science  of 
Science  Policy. 


•  An  NSTC  Working  Group  should  regularly  perform 
portfolio  analyses  of  the  full  spectrum  of  SoSP  across  the 
Federal  government  and  provide  the  President's  Science 
Advisor  with  their  results.  These  analyses  are  tentatively 
described  in  this  report  as  a  "Federal  Innovation 
Framework,"  which  analyses  how  Federal  investments 
and  policy  decisions  affect  the  Nation's  system  of 
innovation. 

•  NSF  and  other  agencies  should  continue  to  support 
the  development  of  a  theoretical  foundation  through 
existing  programs  of  investigator-initiated  research. 
Workshops  and  informational  websites  can  facilitate  that 
dialog. 


"Policymakers  must  often  decide 
whether  to  make  a  choice  on  a 
current  assessment  of  the  costs 
and  benefits  of  taking  action 
based  on  imperfect  information  or 
to  await  additional  scientific  and 
technical  information.  Moreover, 
while  scientific  knowledge  and 
technological  development  is 
changing  constantly,  the  same  is 
not  always  true  of  public  policy. 

As  a  result,  policies  developed 
a  number  of  years  ago  may  not 
reflect  the  latest  scientific  and 
technological  knowledge." 

Stine,  D.  D.  Science  and  Technology 
Policymaking:  A  Primer.  Congressional 
Research  Service,  April  2008. 
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•  Individual  agencies  should  work  together  to  identify  a  core  suite  of  ways  to  measure  and  describe 
technology  adoption  and  diffusion.  They  should  also  develop  ways  in  which  the  many  scientific 
communities  of  practice  for  each  agency  could  be  described  and  analyzed.  Working  subgroups  such 
as  these  would  be  responsible  generating  a  report  to  the  larger  Working  Group. 

•  Federal  agencies  could  work  with  international  counterparts  to  develop  a  consistent  approach  to 
the  Science  of  Science  Policy  that  transcends  national  boundaries,  potentially  through  the  OECD  or 
international  meeting  symposia. 


DEFINITION  -  FEDERAL  INNOVATION  FRAMEWORK: 

Analyses  of  the  scientific  theories  on  science  and  technology  policy  decision  making  that 
explain  how  federal  investments  and  policy  decisions  affect  the  nation's  system  of  innovation, 
and  provide  an  empirical  framework  to  understand  the  Nation's  science  and  engineering 
enterprise.  It  includes  ongoing  evaluation  of  the  relevant  tools  and  metrics  utilized  by 
different  Federal  Agencies.  The  Federal  Innovation  Framework  group,  led  by  OSTP,  will 
include  feedback  loops  that  analyze  the  impact  of  various  policy  instruments.  This  will 
require  cooperation  with  a  wide  variety  of  Federal  agencies,  including  the  Department  of 
Treasury,  DOC  and  the  major  Federal  statistical  agencies  to  understand  the  effects  of  tax 
policy,  labor  policy  and  other  Federal  efforts  that  impact  science  policy. 


Theme  2:  Investing  in  Science  and  Innovation 

The  pragmatic  reality  facing  Federal  agencies  is  that  the  resources  available  for  investing  in  research  are 
limited.  Each  agency,  and  each  program  within  an  agency,  either  explicitly  or  implicitly,  makes  decisions 
about  the  allocation  of  those  resources  on  an  ongoing  basis.  The  ITG's  review  revealed  that  agencies  use 
different  methods  and  tools  to  make  those  allocation  decisions.  Federal  agencies  have  developed  budget 
support  processes  that  enable  them  to  make  investment  decisions  on  an  intra-agency  basis,  but  the  increasing 
complexity  of  the  science  conducted  by  those  agencies  is  becoming  daunting.  Nl FH,  for  example,  must  manage 
a  budget  process  that  supports  27  separate  institutes,  while  the  DOE  manages  a  budget  process  that  includes 
nine  major  R&D  offices  that  invest  in  a  varied  and  complex  range  of  sciences  and  technology.  The  problem  is 
compounded  when  determining  the  benefits  of  investments  across  agencies  or  national  boundaries. 

Question  4:  What  Is  The  Value  Of  The  Nation's  Public  Investment  In  Science? 

There  is  a  wide  body  of  scientific  and  technical  knowledge  created  solely  through  Federal  research  and 
development  investment,  which  would  not  typically  be  sought  through  private  investment  (i.e.  generation  of 
this  knowledge  does  not  have  an  immediate  or  obvious  "payoff").  Value,  in  the  Federal  context  context,  is  thus 
twofold,  and  refers  to  both  the  value  of  the  knowledge  produced  by  governmental  efforts  (how  much  is  the 
knowledge  worth?),  as  well  as  the  value  of  developing  that  knowledge  through  governmental  efforts  (what  is 
the  value  of  learning  this  with  public  funds?).  Frequently,  these  values  are  not  known  until  viewed  in  historical 
or  anecdotal  context.  In  order  to  make  more  informed  and  prioritized  research  investments.  Federal  agencies 
have  a  need  to  better  understand  the  value  of  the  knowledge  likely  to  be  produced  from  their  research 
investments  in  real  time.  Failure  to  do  so  has  very  real  consequences.  For  example,  the  famous  observation  of 
Nobel  Laureate  Bob  Solow:  "You  can  see  computers  everywhere  except  in  the  productivity  statistics",  turned 
out  to  be  due  to  a  misinterpretation  of  data.  This  not  only  led  to  an  understatement  of  U.S.  economic  growth, 
but  also  to  a  fundamental  misunderstanding  of  the  importance  of  information  technology  in  contributing  to 
growth.13 
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There  are  many  methods  of  valuing  publicly  funded  knowledge,  but  most  of  them  rely  solely  upon  qualitative 
assessments,  such  as  committee  reviews  and  case  studies.  Various  agencies,  however,  have  begun  efforts 
to  develop  complementary  quantitative  methods  designed  to  address  the  value  question  within  their  own 
context.  For  example,  DOE  has  begun  to  develop  risk  assessment  and  modeling  tools  that  could  provide 
insights  into  the  value  of  a  mixed  portfolio  of  energy  efficiency  investments.14  Other  efforts  developed  by 
agencies  include  measuring  patents,  citations,  prototype  products  and  processes,  business  finance  measures, 
as  well  as  collaborative  relationships  formed  and  publications.15 16  NIH  counts  the  reductions  in  the  direct  and 
indirect  costs  of  illness,  as  well  as  reductions  in  intangible  costs  due  to  increases  in  longevity  in  better  quality 
of  life.  In  addition,  NSF's  SciSIP  program  has  funded  researchers  who  are  attempting  to  document  the  benefits 
from  publicly  funded  international  collaboration  in  bio-fuels,  as  well  as  the  contributions  of  foreign  graduate 
students  and  post-doctoral  students  to  knowledge  creation  and  diffusion. 

Finding :  Although  determining  the  value  of  publicly  funded  knowledge  is  a  critical  outcome  measure  for  Fed¬ 
eral  science  agencies,  the  analysis  is  largely  agency  specific.  Many  of  the  tools  are  of  uneven  quality  and  the 
broader  discourse  is  often  anecdotal.  Many  open  research  questions  remain.  For  example,  is  it  possible  to  de¬ 
velop  a  full  systems  approach  to  mapping  science,  technology,  and  innovation?  Is  it  possible  to  put  in  place  a 
complete  accounting  of  intangible  assets  and  their  contributions  to  science  and  technology  outcomes  to  create 
an  overall  scientific  measure  of  return  on  investment?  Flow  can  the  community  develop  more  inclusive  mea¬ 
sures  to  capture  the  spillover  effects  between  scientific  discovery  and  technological  innovation,  particularly 
among  universities,  companies,  and  government  laboratories? 

The  creation  of  a  Federal  Innovation  Framework,  as  proposed  here,  could  provide  a  forum  within  which 
varied  community  practices  can  be  shared.  This  Framework  group  would  stimulate  dialog  promoting  a  better 
understanding  of  which  decision  support  tools  could  be  used  by  different  agencies,  allow  for  joint  data 
collection  efforts,  and  stimulate  more  rigorous  methods  of  analyzing  the  scientific  process  among  different 
agencies. 

Question  5:  Is  It  Possible  To  "Predict  Discovery"? 

It  is  extremely  unlikely  that  any  single  model  could  predict  particular  discoveries  and  any  attempt  to  build 
such  a  model  should  be  looked  upon  with  great  skepticism.  New  advances  in  agent  based  modeling  and  an 
increased  capacity  to  simulate  different  scenarios,  however,  hold  the  promise  that  a  series  of  possible  future 
scenarios  could  be  developed.  With  improved  models  of  the  processes  that  lead  to  the  diffusion  of  knowledge 
and  the  evolution  of  the  communities  of  science,  new  and  emerging  areas  of  discovery  could  be  identified 
and  targeted  for  accelerated  assistance.  For  example,  researchers  have  begun  to  look  at  gaps  in  the  "Idea 
Innovation  Network".  The  Idea  Innovation  Network  divides  research  activities  into  six  different  innovation 
arenas  and  their  relatedness  within  a  particular  industrial  sector  (see  box  below).  Gaps  in  our  understanding 
of  this  include  the  roles  of  low-  versus  high-risk  science,  as  well  as  small  versus  large-scale  science.17 

A  similar  effort  has  gone  into  developing  a  complex  systems  model  of  technological  evolution,  focusing  on 
low-carbon  energy  technologies.  Many  agencies,  such  as  DOE,  NIFI,  and  NSF  are  funding  new  tools  that  capture 
data  on  citations  and  patents  which  can  reveal  areas  of  emerging  innovation  activity  and  the  movement 
to  market,  offering  tantalizing  hints  of  real-time  prediction  of  near-term  discoveries.  USDA  Cooperative 
State  Research,  Education,  and  Extension  Research  (CSREES)  and  Forest  Service  (FS)  are  using  logic  models, 
which  describe  work  inputs  and  outputs  within  an  organization,  to  plan  and  assess  their  R&D  portfolios 
and  programs.  Further,  CSREES  is  requiring  that  logic  models  be  included  in  proposals  submitted  to  many 
of  its  programs.  USDA  Economic  Research  Service  (ERS)  is  using  environment  scanning,  commonly  used  by 
companies  to  gain  factual  and  subjective  information  on  business  environments  they  are  considering  entering, 
to  poll  external  stakeholders  on  future  program  directions.  NASA  uses  decadal  surveys  and  strategic  roadmaps 
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to  plan,  relying  heavily  on  independent  assessments,  thereby  identifying  and  prioritizing  promising  scientific 
goals,  missions,  and  programs. 

Finding:  Agencies  are  using  very  different  approaches  and  tools  designed  to  develop  scenarios  that  anticipate 
the  effects  of  discovery  and  innovation.  Many  agencies  are  not  doing  this  at  all.  There  is  very  little  commu¬ 
nication  across  agencies,  and  little  evaluation  of  the  strengths  or  benefits  of  different  approaches.  No  agency 
reported  using  approaches  similar  to  those  developed  by  the  Federal  Reserve,  which  expended  considerable 
effort  to  create  a  complex  econometric  model  based  on  the  best  available  models,  data,  and  tools  to  under¬ 
stand  the  impact  of  different  interventions  on  a  complex  set  of  outcomes.18  In  addition,  there  is  little  transpar¬ 
ency  in  the  analytical  process. 


DEFINITION:  I  DEA  I  NNOVATI  ON  NETWORKS 

These  networks  exist  at  the  level  of  an  industrial  sector  and  market  sector,  and  each  network 
has  six  different  functional  arenas  in  which  various  types  of  innovative  processes  occur.  The  six 
research  arenas  are  basic  research,  applied  research,  research  about  product  development, 
research  on  manufacturing  processes,  research  on  quality  control,  and  research  about  the 
commercialization  and  marketing  of  products.  Each  of  these  functional  arenas  has  its  own 
highly  trained  workers,  dedicated  research  funds,  and  specific  outputs.  An  idea  innovation 
network  is  defined  as  the  research  activities  in  each  of  the  six  arenas  and  the  connectedness 
within  and  among  these  arenas  in  a  particular  industrial  sector. 

Hage,  Jerald  and  J .  Roger  Hollingsworth.  2000.  "A  Strategy  for  the  Analyses  of  Idea 
Innovation  Networks  and  I nstitutions"  Organization  Studies  (Special  Issue:  The 
I nstitutional  Dynamics  of  I nnovation  Systems)  21(5):  971-1004. 


Question  6:  Is  It  Possible  To  Describe  The  Impact  Of  Discovery  On  Innovation? 

The  current  state  of  the  art  in  describing  the  impact  of  discovery,  typically  cast  as  R&D  or  technology,  rests  on 
the  results  of  econometric  studies,  surveys,  case  studies,  and  retrospective  analyses. 

Econometric  studies  include  the  macroeconomic  growth  models  pioneered  by  Robert  Solow19,  which  show 
the  equilibrium  growth  path  for  an  economy  with  an  assumed  endowment  of  technology,  but  pay  little 
attention  to  the  development  of  technology  or  how  it  gets  used  (innovation).  The  Solow  models  have  been 
embellished  and  expanded  by  others20-21,  however,  these  models  are  limited  by  their  treatment  of  discovery 
as  an  exogenous  "black  box"  rather  than  as  part  of  a  larger  ecosystem  of  innovation.  This  kind  of  aggregated 
approach,  which  has  been  implemented  by  the  European  Union,22  is  not  flexible  enough  to  capture  the 
complexity  of  the  feedback  mechanisms  which  the  nascent  literature  indicates  lies  at  the  core  of  innovation. 

Surveys  are  heavily  used  by  a  number  of  agencies.  EPA  relies  on  a  combination  of  partner  surveys,  research 
citations  in  regulatory  and  other  documents,  and  bibliometric  analyses  to  inform  their  broader  program 
reviews  that  assess  research  impact.  Other  agencies  also  track  patents  and  papers.  USDA-CSREES  tracks 
publications  of  papers  that  result  from  its  investments,  as  well  as  patents.23  DOE  has  been  exploring  the  use  of 
a  number  of  stochastic  and  linear  models  to  understand  the  impact  of  scientific  discovery. 

Some  agencies,  such  as  NIST,  prospectively  estimate  their  impacts  through  standard  benefit-cost  analysis. 
Others,  such  as  the  CDC,  conduct  case  studies  summarizing  selected  research  projects.24  The  CDC  has  also 
used  retrospective  bibliometric  analysis  in  the  past,  and  is  now  moving  to  an  internet-based  information 
tracking  system.  Most  agencies  (NIH,  DOE,  NASA,  EPA,  USDA)  track  milestones  and  use  peer-review,  advisory 
committees,  and  survey  instruments  to  assess  impacts.  Other  agencies  such  as  USGS  have  explored  the 
importance  and  value  of  improved  scientific  information  in  land  use  decision  making.  A  few  agencies,  such  as 
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DOE  and  NIH,  have  begun  to  experiment  with  dynamic  modeling  and  options  modeling  as  ways  to  describe  the 
impact  of  discovery. 

Finding :  Agencies  are  using  a  wide  variety  of  approaches  to  describe  the  impact  of  discovery.  However,  new 
approaches  are  being  developed  by  the  academic  community  that  utilize  new  tools  such  as  science  mapping 
(correlating  funding  with  research  outputs),  and  new  datasets  such  as  the  OECD  international  database  of 
inter-organizational  collaborative  agreements.  The  Federal  community  still  lacks  a  theoretical  framework  that 
it  can  use  to  assess  the  impact  of  science  and  technology  policies  on  discovery  and  resultant  social  welfare 
outcmes. 


DEFINITION  -  SCIENTIFIC  DISCOVERY: 

Scientific  discovery  may  be  described  as  the  observation  of  new  phenomena,  new  actions, 
or  new  events  and  providing  new  reasoning  to  explain  the  knowledge  gathered  through 
such  observations  with  previously  acquired  knowledge  from  abstract  thought  and  everyday 
experience.  In  scientific  research,  exploration  is  one  of  three  purposes  of  research,  the 
other  two  being  description  and  explanation.  Discovery  is  made  by  providing  observational 
evidence  and  attempts  to  develop  an  initial,  rough  understanding  of  some  phenomenon. 

htto://en.  Wikipedia.  ora/wiki/Discoverv_  %  28observation%  29 


Question  7:  What  Are  The  Determinants  Of  Investment  Effectiveness? 

Federal  agencies  are  accountable  for  the  effectiveness  of  their  R&D  investments.  This  requirement  has  been 
made  explicit  by  OMB  through  the  use  of  R&D  Investment  Criteria  and  the  PART.  As  a  result,  some  agencies 
have  developed  a  variety  of  tools  to  assess  the  effectiveness  of  their  investments,  while  others  still  lack  such 
tools.  In  terms  of  assessing  the  success  of  individual  projects,  agencies  have  made  tremendous  strides. 


Assessing  R&D 
Benefits  &  Effectiveness 
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Different  agencies  use  different  tools  to  make  policy  and  investment  decisions  -  the  method  depends 
on  the  scope  and  size  of  the  science  involved.  Types  of  Evaluation  Methods  for  Assessing  R&D 
Benefits  and  Effectiveness  for  Various  Scales  of  R&D  Investment,  U.S.  Department  of  Energy. 
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Finding:  Techniques  used  by  Federal  agencies  to  determine  program  effectiveness  span  the  spectrum  from 
those  in  the  pilot  stages  to  those  that  are  mature.  A  list  of  these  approaches  as  revealed  by  the  SOSP  ITG 
questionnaire  include: 

•  Growth  Accounting—  used  to  better  generate  estimates  of  the  Nation's  productivity  performance  in 
terms  of  contributing  factors  and  outputs. 

•  Knowledge  Economy—  these  composite  knowledge  indicators  are  used  to  improve  investment 
decisions  for  R&D,  education,  and  capital  resources. 

•  Financial  Reporting— these  reports  are  used  to  provide  a  balanced  scorecard  of  physical  as  well  as 
intangible  assets. 

•  Valuation  of  Innovation— business  executives  and  financial  markets  (to  better  value  R&D  activity 
and  related  intangibles)  estimate  financial  results,  improve  long  term  stock  market  valuations,  and 
predict  outcomes. 

•  System  Dynamics— ex pand  the  range  of  "real-time"  innovation  metrics  to  help  build  more  robust 
system  dynamics  models  and  policy  simulations. 

•  General  Purpose  Technology  (GPT)  —  improved  analysis  of  the  strategic  contribution  of  GPT's  may 
set  the  stage  for  incremental  innovation  and  have  the  inherent  potential  for  pervasive  application  in 
a  wide  variety  of  industries 

•  Tech-led  Regional  Development  and  Clusters— used  to  shift  the  emphasis  from  strengthening  inputs 
to  the  innovation  infrastructures  toward  improving  the  efficiency,  rate,  and  output  of  innovation.25 

While  these  tools  are  extremely  useful  for  assessing  program  effectiveness,  the  evaluation  of  complex 
portfolios  (such  as  those  managed  by  NSF,  DOE  or  NIH)  remains  difficult.  Agencies  are  keenly  aware  of  the 
deficiencies  in  their  approaches.  One  agency,  when  responding  to  the  ITG's  questionnaire,  noted:  "Waiting  to 
count  publications,  for  example,  is  too  late  to  affect  the  real-time  assessment  of  a  particular  grant."  Another 
noted:  "We  need  a  great  deal  more  outcome  data.  The  data  currently  available  are  inadequate  for  program 
evaluation  purposes.  Resources  are  needed  to  improve  Federal  and  other  data  collections  systems". 

Of  course,  evaluation  approaches  vary  even  within  an  agency,  depending  on  the  nature  of  the  needs,  as  the 
Figure  demonstrates.  For  details,  see  the  "Current  and  Potential  Toolkit  For  Science  and  Innovation  Policy" 
section. 

Theme  2  Recommendations 

The  ITG  recommends  that  a  Federal  Innovation  Framework,  as  described  in  Theme  1,  be  created  to 
accelerate  the  development  of  the  nascent  community  of  science  policy  researchers  and  practitioners 
through  interagency  coordination  and  targeted  investments.  Led  by  OSTP,  this  working  group  could  assist 
each  agencies'  specific  science  policy  analysis  capabilities,  while  coordinating  efforts  to  identify  and  promote 
best  practices  and  information  sharing  around  science  policy  and  R&D  investment  management.  Specific 
recommendations  include: 
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•  Agencies  (such  as  NIH,  NSF,  DHS,  CDC,  VA,  NASA,  USGS,  and  DOE)  should  work  together  to  develop 
a  pilot  data  infrastructure  that  captures  key  data  about  their  respective  scientific  communities.  The 
development  would  include  an  assessment  of  the  validity  of  the  new  visualization  techniques  to 
describe  the  changing  structure  of  science. 

•  Agencies  should  work  together  to  develop  pilot  standards  for  identifying  ways  of  measuring  the 
value  of  knowledge,  which  could  then  be  adapted  to  the  missions  of  individual  agencies. 

•  Agencies  should  work  together  to  develop  standard  approaches  for  using  bibliometrics  to  assess 
science  impact. 

The  academic  research  community  should  continue  to  be  supported  to  perform  the  best  research  to  develop 
new  analytical  tools,  methods,  and  metrics  to  support  the  emerging  science  of  science  policy. 

Theme  3:  Using  the  Science  of  Science  Policy  to  Address  National  Priorities 

By  developing  the  core  data,  models,  and  tools  that  will  be  necessary  to  answer  the  10  Scientific  Challenges 
addressed  in  this  Roadmap,  significant  advances  in  economics, 
sociology,  psychology  and  political  science  could  be  made  that 
would  benefit  all  of  society. 

Advances  in  the  social  and  economic  sciences  provide  the 
foundation  for  the  science  of  science  policy  and  should  be 
leveraged  to  develop  decision  support  tools  that  policy  makers 
could  use  when  grappling  with  extraordinarily  complex  national 
challenges.  Decisions  to  invest  in  science  are,  by  necessity,  made 
in  the  context  of  other  investment  decisions  (such  as  investments 
in  defense  and  transportation  infrastructure),  and  the  current 
absence  of  appropriate  analytical  tools  limit  the  careful  analysis  of 
the  relative  costs  and  benefits  of  such  investments.  The  absence 
of  tools  stands  in  sharp  contrast  to  the  intricate  econometric 
models  used  by  the  financial  community  to  understand  trade  and 
industrial  phenomena.  These  econometric  tools  have  developed 
over  decades  and  are  just  now  reaching  maturity  as  the  field 
of  economics  has  matured;  in  similar  manner,  it  is  expected  that  tools  and  models  for  science  policy  should 
develop,  as  the  field  matures. 

The  development  of  a  Federal  Innovation  Framework  could  be  used  not  only  to  assess  the  state  of  the  art 
in  SoSP  and  its  relevant  tools  as  described,  it  could  also  be  utilized  to  perform  analyses  relevant  to  national 
priorities.  Examples  include  addressing  climate  change  technology  options,  agricultural  policy  that  is  impacted 
by  the  decision  to  pursue  biofuels,  and  understanding  the  implications  of  improved  health  care  in  the  U.S.  on 
social  support  networks  such  as  Medicare  and  Medicaid. 


The  Fluman  Genome  Project  was  a 
concerted  Federal  effort  in  science  that 
resulted  in  interagency  cooperation  to 
solve  a  national  challenge. 

U.S.  Department  of  Energy, 
http://aenomics.enerav.gov/ 


DEFINITION  -  INNOVATION  INFRASTRUCTURE 

Innovation  infrastructure  is  the  physical  and  policy  infrastructure  that  supports  innovators. 
Innovation  infrastructure  includes  information  networks,  intellectual  property  protections, 
business  regulations,  and  structures  for  collaboration  among  innovation  stakeholders.  These 
supporting  infrastructures  should  be  adaptive  as  the  needs  for  scientific  innovation  evolve, 
requiring  informed  policy  that  evolves  apace. 

Innovate  America,  National  Innovation  Initiative  Report,  Council  on  Competitiveness,  2005, 
http://www.compete.ora/imaaes/uploads/File/PDF%20Files/NI  l_ I  nnovate_America.pdf. 
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Question  8:  What  Impact  Does  Science  Have  On  Innovation  And  Competitiveness? 

Vannevar  Bush,  in  his  seminal  1945  treatise,  "Science, 

The  Endless  Frontier,"  wrote:  "New  products  and  new 
processes  do  not  appear  full-grown.  They  are  founded  on 
new  principles  and  new  conceptions,  which  in  turn  are 
painstakingly  developed  by  research  in  the  purest  realms 
of  science."  Bush's  treatise  helped  shape  U.S.  Federal 
government  policy  on  science  and  technology  and  gave 
great  support  for  the  notion  that  innovation  in  science 
and  technology  is  a  key  driver  of  U.S.  national  economic 
prosperity  and  national  security. 


In  the  last  half  century,  economists  and  social  scientists 
have  attempted  to  document  the  impact  of  the  scientific 
enterprise  on  U.S.  innovation  and  competitiveness,  with 
mixed  results.  At  the  macro  level,  economists  such  as 
Robert  Solow  (who  won  the  Nobel  Prize  in  Economics 
for  his  work  on  the  impact  of  innovation  on  economic 
growth)  have  developed  widely  accepted  theories  that 
explain  how  investments  in  science  and  technology 
have  a  positive  impact  on  economic  growth.  There  are 
disputes  about  the  rate  of  growth  and  other  factors, 
but  there  is  no  dispute  that  the  impact  is  positive.  At 
the  micro  level,  economists  such  as  Erik  Brynjolfsson 
have  documented  the  positive  impact  of  information 
technology  on  organizations  and  firm  performance1.  This 
level  of  analysis  is  helpful  because  it  provides  comfort  to  Federal  policy  makers  that  their  investment  strategy  is 
sound,  but  macro  level  econometric  models  cannot  answer  the  basic  question,  how  much  funding  is  enough ? 


Vannevar  Bush,  President  Science  Advisor, 
helped  establish  the  science  and  technology 
policy  that  enabled  the  U.S.  Federal 
government  to  play  a  central  role  in  the 
creation  of  the  world's  leading  innovation 
economy. 

http://www.carneaieinstitution.org  Yearbook 


For  example,  policy  makers  do  not  have  rigorous  and  quantitative  tools  that  explain  how  various  technology 
streams  could  reduce  U.S.  energy  consumption,  or  whether  or  not  investments  in  rare  diseases  that  afflict 
small  populations  are  more  important  to  society  than  investments  in  common  diseases,  such  as  H IV  or  cancer, 
that  require  long-term  investments  with  uncertain  prospects  for  success. 

There  is  a  considerable  literature  on  impact  evaluation  in  the  social  sciences  that  is  related  to  this  problem 
set  confronted  by  the  science  of  science  policy  community.  There  are  also  new  developments  in  the  ways 
of  collecting  data  on  complex  inputs,  such  as  the  generation  of  ideas  through  tracking  citations,  as  well  as 
collecting  information  on  the  transmission  of  ideas  through  social  networks  and  scientific  communities.  The 
increasing  availability  of  administrative  data,  combined  with  advances  in  privacy  protection,  could  allow  for  the 
examination  of  the  impact  of  different  types  of  tax  credits  on  businesses,  or  illuminate  hiring  patterns. 

Finding:  The  ITG  finds  that  there  is  a  real  opportunity  to  develop  new  tools  and  data  sets  that  could  be  used 
to  quantify  the  impact  of  the  scientific  enterprise  thus  far  on  innovation  and  competitiveness.  These  impacts 
could  include  the  generation  of  knowledge,  the  health  of  the  universities  receiving  funding,  the  growth  of  the 
STEM  workforce,  or  the  growth  and  survival  of  those  businesses  and  their  workforces  most  closely  linked  to 
the  scientific  enterprise. 


1  See,  for  example,  Aral,  S.,  Brynjolfsson,  E.  and  Wu,  D.J.,  Which  Came  First,  IT  or  Productivity?  The  Virtuous  Cycle  of  Investment  and 

Use  in  Enterprise  Systems  ,  MIT  Center  for  Digital  Business  Working  Paper,  (October,  2006). 
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Question  9:  How  Competitive  Is  The  U.S.  Scientific  Workforce? 

The  dominance  of  the  U.S.  scientific  enterprise  is  dependent  on  the  quality  of  its  scientific  workforce.  Our 
national  competitiveness  in  science  has  been  enhanced  by  the  many  foreign  students  who  come  to  U.S. 
universities,  as  well  as  by  the  many  skilled  STEM  workers  who  move  to  the  U.S.  from  other  countries. 

However,  U.S.  scientific  dominance  may  be  threatened  by  the  ability  of  other  countries  to  train  their  own 
technical  workforce,  as  well  as  the  ability  of  businesses  to  tap  into  the  innovative  capacity  of  a  global  scientific 
community  and  encourage  technical  workers  from  the  U.S.  to  relocate  to  nations  such  as  China  and  India 

There  has  been  considerable  policy  debate  about  the  quality  of  the  STEM  workforce  but  arguments  supporting 
either  side  of  the  issue  are  hindered  by  a  lack  of  longitudinal  data  on  the  post  secondary  STEM  workforce. 
Questions  about  this  workforce  include:  what  happens 
to  them  when  they  graduate;  where  do  they  get 
jobs;  and  what  are  their  labor  market  trajectories? 

Industries  accustomed  to  the  free  flow  of  technical 
workers  sometimes  find  that  national  security 
concerns  impede  their  ability  to  recruit  and  retain 
international  workers.  Universities  are  now  training 
large  numbers  of  foreign  scientists  who  return  to  their 
home  countries. 

In  addition,  few  direct  measures  of  the  global  science 
and  engineering  (S&E)  labor  force  exist.  Recent 
analytical  work  has  been  done  in  the  United  States 
and  Europe  on  "network  analysis,"  which  could  be 
used  to  measure  the  impacts  of  and  reasons  for  the 
"high-skilled  diaspora"  that  has  been  discussed  in  the 
science  and  engineering  literature.  Developing  and  extending  the  results  of  those  network  analysis  studies 
could  go  a  long  way  toward  answering  questions  about  labor  market  formation  in  S&E. 

Finding:  Many  critical  questions  about  the  quality  and  global  nature  of  the  STEM  workforce  cannot  be  an¬ 
swered  due  to  a  lack  of  data.  While  the  models  and  tools  exist  to  study  flows  of  workers  within  and  across 
disciplines  and  nations,  lack  of  data  means  that  the  science  policy  community  cannot  answer  important  ques¬ 
tions  about  the  scientific  enterprise. 

Question  10:  What  Is  The  Relative  Importance  Of  Different  Policy  Instruments  In  Science  Policy? 

The  primary  policy  instrument  that  the  U.S  Federal  government  wields  for  science  policy  is  broad  Federal 
investment  in  key  areas  of  science  and  technology.  As  a  result,  the  U.S.  has  invested  trillions  of  dollars  over 
six  decades  in  university  research,  scientific  workforce  development,  national  laboratory  infrastructure,  major 
scientific  instruments  and  other  areas.  This  has  resulted  in  an  unrivaled,  world  class  scientific  infrastructure 
that  will  be  extremely  difficult  for  any  other  nation  to  duplicate.  This  massive  investment  in  science  and 
technology  by  the  Federal  government  is  perhaps  the  key  U.S.  competitive  advantage  in  the  emerging 
international  innovation  competition.  This  competitive  advantage  has  not  gone  unnoticed,  which  helps 
explain  the  drive  by  China,  Korea  and  other  Asian  countries  to  make  major  national  investments  in  science  and 
technology  and  the  decision  by  the  European  Union  to  triple  its  rate  of  investment  in  science  and  technology. 

Investment  in  science  and  technology,  however,  is  only  one  of  the  policy  instruments  available  to  science  policy 
makers;  others  include  fostering  the  role  of  competition  and  openness  in  the  promotion  of  discovery,  the 
construction  of  intellectual  property  systems,  tax  policy,  and  investment  in  a  STEM  workforce.  However,  the 


Students  in  the  lab  -  workforce  development,  NSF. 
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probable  impact  of  these  various  policies  and  interventions 
is  largely  unknown.  This  lack  of  knowledge  can  lead  to 
serious  and  unintended  consequences.  For  example. 
Federal  efforts  to  increase  industrial  innovation  through 
university  technology  transfer  programs  has  led  to  concerns 
that  universities  are  now  focusing  too  much  on  near-term 
research  and  too  little  on  their  traditional  strength,  which 
is  long-term  and  basic  research.  The  recent  doubling  of 
investment  at  N I FH  has  created  concerns  within  the  medical 
community  about  the  impact  these  investments  have 
had  on  the  production  of  PhDs  and  tenured  faculty  at 
universities. 

Finding:  There  has  been  very  little  investment  in  the  study 
of  alternative  science  policy  instruments  either  in  the 
United  States  or  in  other  countries.  While  the  models  and 
tools  exist  to  examine  the  effectiveness  of  different  ap¬ 
proaches,  there  are  gaps  in  the  analytical  structure,  the 
data  infrastructure,  and  in  the  ways  in  which  information 
can  be  conveyed  to  policy  makers. 

Theme  3  Recommendations 

The  ITG  recommends  investing  in  data  collection,  analytical 
tools,  and  ways  to  present  complex  information. 
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•  Several  core  datasets  should  be  established  and  made  available  to  both  the  research  and  policy 
community. 


•  The  first  of  these  is  a  longitudinal  dataset  on  businesses,  oversampling  those  businesses 
which  are  critical  to  American  competitiveness,  such  as  high  tech,  biotech,  and 
multinational  firms.  Existing  administrative  data  should  be  used  to  keep  the  costs 
manageable  and  the  sample  scientific. 

•  The  second  is  a  longitudinal  dataset  on  the  STEM  workforce.  Although  the  original 
dataset  could  have  a  survey  basis,  every  effort  should  be  made  to  exploit  longitudinal 
administrative  records,  and  to  partner  with  other  countries  in  order  to  capture  the  long 
term  dynamic  adjustments  of  workers. 


•  The  link  between  workers  and  firms  must  be  tracked  through  administrative  records  and 
other  modes  of  data  collection  so  that  the  relationship  between  the  humans,  who  are  the 
sources  of  innovative  ideas,  and  the  firms,  who  bring  the  ideas  to  market,  can  be  analyzed. 

•  The  Federal  Innovation  Framework  must  include  feedback  loops  that  analyze  the  impact  of  various 
policy  instruments.  This  will  require  cooperation  with  a  wide  variety  of  other  Federal  agencies, 
including  the  Department  of  Treasury,  the  Census  Bureau  and  the  major  Federal  statistical  agencies 
to  understand  the  effects  of  tax  policy,  labor  policy  and  other  Federal  efforts  that  impact  science 
policy. 
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Current  and  Potential  Toolkit  For 
Science  and  Innovation  Policy 


An  obvious  question  is:  "What  toolkit  of  models,  tools  and  metrics  is  currently  available  for  policy  makers 
when  making  science  policy  decisions,  and  what  could  be  available  with  additional  focus  on  the  development 
of  SoSP?"  In  order  to  answer  this  question,  the  ITG  relied  on  the  literature  synthesis,  the  questionnaire,  and 
its  own  experience.  It  first  identified  the  models,  tools  and  metrics  currently  being  used  by  Federal  agencies, 
which  included: 


Models  and  Tools: 

Quantitative  Analysis 

•  Deterministic  Models:  Econometric;  Risk  Modeling;  Options  Modeling;  Cost 
Benefit;  Cost  Effectiveness 

•  Stochastic  Models:  Agent  Based;  System  Dynamics 

Qualitative  Analysis 

•  Case  Studies;  Peer/Expert  Review;  Delphi;  Strategic/Logic 

Visualization  Tools 

•  Network  Analysis;  Visual  Analytics;  Science  Mapping;  Scientometrics 

Data  Collection  Tools 

•  Survey;  Web  Scraping;  Administrative  Data;  Data  Mining 

Metrics: 

Outcome 

•  Scientific/Micro  Level:  Innovation;  Competitiveness;  Knowledge  Increase 

•  Program/Portfolio:  Effectiveness;  Value 

•  Systems  Level:  Productivity;  Quality  of  Life;  Workforce  Characteristics;  GDP 

Budget  and  Performance 

•  Earned  Value;  Process  Metrics;  Efficiency;  Marginal  Cost 

Inputs 

•  Bibliometrics:  Citations;  Patents;  Scientific  Papers 

•  Community/Network:  Network  Value;  Effectiveness;  Structure;  Workforce 

The  ITG  then  identified  the  dimensions  along  which  the  methods,  tools  and  metrics  had  value  for  science 
policy  as  well  as  the  dimensions  for  assessing  the  potential  cost  if  any  investment  were  to  be  required  to 
bring  them  to  full  use.  Five  criteria  were  identified,  three  associated  with  the  potential  value  of  the  element: 
Relevance  for  Science  Policy,  Breadth  of  Use,  Scientific  Rigor;  and  three  associated  with  the  potential  cost: 
Maturity  of  the  Method  or  Tool,  Availability  and  Quality  of  Data  Required. 

The  degree  to  which  the  element  provides  a  significant  contribution  to 
resolving  one  or  more  of  the  10  Scientific  Challenges  identified  by  the  ITG. 
The  extent  of  the  adoption  of  the  element  in  the  Federal  or  academic 
science  policy  context. 

The  quality  of  the  scientific  foundation  of  the  element,  in  terms  of 
publications,  scientific  openness,  size  of  community  and  reproducibility. 

The  degree  to  which  the  element  is  used  in  the  Federal  or  academic  science 
policy  context. 

The  practicality  of  using  the  element  to  develop  the  empirically  based 
platform  for  decision  making  that  is  the  goal  of  science  policy. 


Relevance  to  Vision: 
Breadth  of  Use: 

Scientific  Rigor: 

Maturity  of  the  Method 
or  Tool: 

Availability  and 
Qualityof  Data  Required: 
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Acronyms 

ATP 

Advanced  Technology  Program  (NIST) 

CERN 

European  Organization  for  Nuclear  Research  (Conseil  Europeen 
pour  la  Recherche  Nucleaire) 

CDC 

Centers  for  Disease  Control  and  Prevention 

AMERICA  COMPETES  ACT 

America  Creating  Opportunities  to  Meaningfully  Promote 

Excellence  in  Technology,  Education,  and  Science  Act,  2007 

COSEPUP 

Committee  on  Science,  Engineering  and  Public  Policy 

CRIS 

Current  Research  Information  System  (USDA) 

CSREES 

Cooperative  State  Research,  Education,  and  Extension  Research 
(USDA) 

DHHS 

U.S.  Department  of  Health  and  Human  Services 

DHS 

U.S.  Department  of  Homeland  Security 

DOC 

U.S.  Department  of  Commerce 

DOD 

U.S.  Department  of  Defense 

DOE 

U.S.  Department  of  Energy 

DOEd 

U.S.  Department  of  Education 

DOI 

U.S.  Department  of  Interior 

DOT 

U.S.  Department  of  Transportation 

EPA 

Environmental  Protection  Agency 

ERS 

Economic  Research  Service 

FS 

Forest  Service  (USDA) 

GPRA 

Government  Performance  Results  Act 

GPT 

General  Purpose  Technology 

IRS 

Internal  Revenue  Service 
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ITG 

Interagency  Task  Group 

NAS 

National  Academy  of  Sciences 

NASA 

National  Aeronautics  &  Space  Administration 

NIH 

National  Institutes  of  Health 

NIST 

National  Institute  of  Standards  and  Technology 

NOAA 

National  Oceanic  and  Atmospheric  Administration 

NSF 

National  Science  Foundation 

NSTC 

National  Science  and  Technology  Committee 

OMB 

Office  of  Management  and  Budget 

OSTP 

Office  of  Science  and  Technology  Policy 

PART 

Program  Assessment  Rating  Tool 

R&D 

Research  and  Development 

ROI 

Return  on  Investment 

S&E 

Science  and  Engineering 

S&T 

Science  and  Technology 

SciSIP 

Science  of  Science  and  Innovation  Policy 
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Social,  Behavioral,  and  Economic  Sciences 

SoSP 

Science  of  Science  Policy 

STEM 

Science,  Technology,  Engineering,  and  Mathematics 

USDA 

U.S.  Department  of  Agriculture 

VA 

U.S.  Department  of  Veterans  Affairs 
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Appendix  A  -  Charter 


National  Science  and  Technology  Council 
Committee  on  Science 

Committee  on  Homeland  and  National  Security 
Subcommittee  on  Social,  Behavioral  and  Economic  Sciences 
Interagency  Task  Group  on  Science  of  Science  Policy 

Charter 


A.  Preamble 

The  Interagency  Task  Group  on  Science  of  Science  Policy  (hereafter  referred  to  as  the  “Task 
Group”)  is  hereby  established  by  the  Subcommittee  on  Social,  Behavioral  and  Economic 
Sciences  (SBE).  The  Task  Group  serves  as  part  of  the  internal  deliberative  process  of  the 
Subcommittee,  which  provides  guidance  and  direction. 


B.  Purpose  and  Scope 

Currently,  science  policy  discussions  are  dominated  by  advocates  for  particular  scientific  fields 
or  missions  and  policy  decisions  are  frequently  based  upon  past  practice  or  data  trends  that  may 
be  out  of  date  or  have  limited  relevance  to  the  current  situation.  We  know  that  past  investments 
in  basic  scientific  research  have  had  an  enormous  impact  on  innovation,  economic  growth  and 
societal  well-being,  but  we  do  not  have  the  capacity  to  predict  how  best  to  make  and  manage 
future  investments  so  as  to  exploit  the  most  promising  and  important  opportunities. 

While  some  fields  benefit  from  the  availability  of  real-time  data  and  computational  models 
which  allow  for  predictive  analyses,  science  policy  does  not  benefit  from  a  similar  set  of  tools 
and  modeling  capabilities.  It  is  imperative  to  advance  the  scientific  basis  of  science  policy, 
through  the  development  of  data  collection,  analyses  and  modeling  tools,  so  that  we  can  make 
future  policy  decisions  based  on  sound  science  and  informed  judgment.  We  must  also  develop 
both  quantitative  and  qualitative  tools  to  enable  the  collection  of  real-time  data  and  to  facilitate 
better  retrospective  analysis  of  the  impact  of  federal  investments  on  scientific  discovery  and 
innovation,  the  economy  and  society.  In  this  way,  we  can  leam  from  past  investments  and  refine 
the  accuracy  of  our  predictive  models. 

hi  order  to  advance  the  academic  discipline  of  the  science  of  science  policy,  the  SBE 
subcommittee  is  establishing  a  Task  Group  that  will  develop  a  roadmap  for  federal  efforts 
directed  toward  the  long-term  development  of  a  science  of  science  policy. 


C.  Objectives 

In  formulating  the  roadmap,  the  Task  Group  will  pursue  the  following  objectives: 

•  Assess  and  inventory  the  current  status  of  Federal  and  international  efforts  in  the  science 
of  science  policy  and  determine  where  gaps  exist. 

•  Determine  the  sources  of  data  and  identify  tools  for  modeling  and  analysis  that  have  the 
potential  to  contribute  to  improved  indicators  and  metrics  for  national  and  international 
research  and  development  (R&D)  investments. 
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•  Identify  and  coordinate  Federal  funding  opportunities  to  develop  tools,  theories  and 
methodologies  that  will  advance  the  science  of  science  policy,  and  recommend  joint 
research,  modeling,  and  evaluation  projects  that  would  enable  Federal  agencies  to 
collaborate,  coordinate  and  leverage  resources  and  efforts. 

•  Develop  linkages  between  these  opportunities  and  the  activities  of  other  groups  and 
agencies  that  are  also  interested  in  understanding  and  modeling  the  national  and  global 
R&D  enterprise  and  in  predicting  and  evaluating  the  impacts  of  R&D  investments, 
highlighting  path  breaking  techniques  that  might  contribute  to  deeper  understanding  and 
marketable  innovations. 

•  Develop  a  roadmap,  that  incorporates  the  results  of  the  government-wide  survey  and, 
where  appropriate,  consultation  with  private  sector  experts.  This  roadmap  would  offer  a 
variety  of  strategies  to  address  the  current  gaps  in  theory,  data,  methodologies,  and 
models. 

•  Periodically  assess  progress,  determine  whether  course  corrections  are  needed  and  submit 
progress  reports  to  the  SBE  subcommittee  on  a  regular  basis. 

D.  Membership 

The  following  agencies  are  represented  on  the  Interagency  Task  Group  on  Science  of  Science 
Policy: 

1 .  Department  of  Energy  (co-chair) 

2.  National  Science  Foundation  (co-chair) 

3.  Department  of  Agriculture 

4.  Department  of  Commerce 

5 .  Department  of  Defense 

6.  Department  of  Education 

7.  Department  of  Health  and  Human  Services 

8.  Department  of  Homeland  Security 

9.  Department  of  the  Interior 

1 0.  Department  of  State 

1 1 .  Department  of  T ransportation 

12.  Department  of  Veterans  Affairs 

13.  National  Aeronautics  &  Space  Administration 

1 4.  National  Institutes  of  Health 

1 5.  National  Institute  of  Standards  and  Technology 

16.  Office  of  Science  and  T echnology  Policy 

1 7.  Office  of  Management  and  Budget 

Other  organizations  may  participate  as  appropriate. 

E.  Interactions  with  Other  Groups 

The  Task  Group  may  interact  with  other  Federal  government  organizations  and  NSTC  bodies  as 
required  in  the  performance  of  its  work.  The  Task  Group  may  also  interact  with  Federal 
advisory  bodies  such  as  the  National  Science  Board  (NSB)  and  the  President’s  Council  of 
Advisors  on  Science  and  Technology  (PCAST).  The  Task  Group  may  interact  with  and  receive 
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ad  hoc  information  and  advice  from  private  sector  groups  as  consistent  with  the  Federal 
Advisory  Committee  Act. 


F.  Termination 

Unless  renewed  by  the  chairs  of  the  SBE  Subcommittee  prior  to  its  expiration,  the  Interagency 
Task  Group  on  Science  of  Science  Policy  shall  terminate  no  later  than  March  31, 2009. 


G.  Determination 

I  hereby  determine  that  the  formation  of  the  Interagency  Task  Group  on  Science  of  Science 
Policy  is  in  the  public  interest  in  connection  with  the  performance  of  duties  imposed  on  the 
Executive  Branch  by  law,  and  that  such  duties  can  best  be  performed  through  the  advice  and 
counsel  of  such  a  group. 


OUt/L  !k  if  kkllw,. 

Josi 

;ph  Kidman,  DHS,  (Co-Chair) 

Subcommittee  on  Social,  Behavioral  and  Economic  Sciences 


-rk ir.bfr  2c,  ZjJOC 
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Apppendix  B  -  Science  Magazine  Editorial 


Editorial 


Wanted:  Better  Benchmarks 

H»  emdi  vk»jIi!  a  anjis  vprtiil  l4  k'aaLf 1  whai  bad  of  'i'  Him  path  fimi  pi  Narc  -.etui* 
F*Ib  lectori?  Den  to  fro&qj  K  *,  is*s  perf-Knim  mpi>  t  shtuiipr  of  fwadjuv 

lt  a  npfeil  lM' perf.r:iM-j'  aod  nelrieel  s..rek*  petiTY  .[iimuci  Lnii  Ili  he  *iLi! ami  aiu»enl 

ItuLi'i  us  a  hiuh.t  iinhie  jdvix'Uiy  .unl-raf  Ihjl  iA«,  ii.  ihjl  are  driers  Lie  i>f  i'.HiR 

i^nisiny  A  prm,  "SdemT  of  WWBPh  pdlFY"  4u=ni^^[i,  jrul  h  rp>  wife  rrtrr+imprUipg'  pidjiue- 
ftl  ifteciUQIB  xriri  ffr  mnrf  rmHIihl#-  idv-ttHk-'Y 

MdrvrlnpeildiJiliEjni.  dmlcpihpinatin^  lo.'jr,  inr i.  trpwdlia  BmllostfytHl  ttdmui.  rJik-Jjici HKl:nmi:k 
« S*py«  «o  hwc  M’Otiiti..  monpiS;  Studies  Cunt;  ilw  |Wu  iSj*  died  S  arama  ji  Itfris  ;levf -ofni-rtii  led  lo 
ip«i6?f  etiwu'ciL  [T«liLiDnry,  md  tor  ietoraHwoi  uidinnio^  p  (wtieuiir,  bn  hre-n  i  rnapr  fcsw  in  uubuhuii  US 
FCiYih  R>.1y  I'li’wt".  flic  i:netfj>n  is  mn  whether  keiiJj  im  ritrerr.i  me  i.-|K'na.n.  but  whn  unedireni  *aalrg.ii  ire 
ni.'d  rll'-Llr.  c  2  torrjpj.il>,  L-haniiiny  gtoul  et>  zuonienl  to  kiazr,  Herr.  iLj.ru,  toirge 

late  to  Jiue  of  to  lee  hr...  L  wndJum  Sharply  Liffciqi  up.zuirjt  raid  reyanLnp  to  pjsnhA.1iuti.unJS  *:jcdtdi 
ha  meet  pnudu  jT'fiT.r.r.;'  thixhiyss  *  Dir  dJTcrnjjni  nur  it.  Ihe  Jiln7urt.Lh.1rn  ol  >1- .  rnuc.-.  ,iu  regst ikr.n  Li; 
iUiJtir. uf  Jiherr  hoiitoi  Jdwtoiri  ill  to  l  .i.lr;!  Sjfiii  ril olhn  aunlr-rt. (to  il.lv. 

Ouru  sml  Irntj-i  b  die  m<n  eounrift.ibr  «n  uf  g^emh  is  die  otiIiht  of  irifflum 

die  ifcjh,  ihhrngji  i.  HLk.  munhe™  air  miuJL  Lrldiwr  In  ceisr  ip  dir  IJnded  fipj*r-i 
kifrnrMa.  to  rurjiri!  pn.tli.tiin:  oTC  &  jltieI  .sHjpccd  ho  nur  low  ,ujj!iul jr. 

:JL-_  1-rrt-j-n  frniphay  r>>  "-ih^l  ih-rl  (emi  lub-Lff'UflimJir..  Cm  pi  JuilJ 

etn  md  pmiiA-i  KHOhm»  ofihu-aiue  toquiirt  a  htvnln  nadafwidiqfof 
’jidueammu.  I.iAi:  .  in  ■  rrnniie  of  cjIjui  Ihjl  wmihL  alto*  in  ln>  .ill j. h 
pEiutohibUa  lL>  rlillrtcrti  lijftrr  S-.f7ie j.  Jnitirci to  .dpeiuiimr 
eKfKim^  wiiih  fcj  felt  of  die  lAiwd  Stow  *ul  doubfets  njllhr  fttun  drur 
to  Trial  I  rr  ca  dewHering  fi.nnnuei  Hrrr.  u  ekewhepe  irvjioUy  ilehae* 
the  beniuTLtij.  itonot  ‘fvii.  to  thmuel’ira 

Use  iLrtj  wr  dfflrtr  la  cuGAt  i!to  i*-  MBtjrl^np  dml  to  fiar.^wurk  Efi 
■beii  *r  uriler jr^  the  leliN^tfl!  tmor?  tuetkT.  pit.  rrrtriniir.  and  uh'ieky. 

Lae  mrUlULEy-  ri-  .  ,n  v  ■  tn  r,  1  ifrir  .  j i  :n  -|  rii m, I  illi  bow-n-d,  LTT^ra 

oil  ittfto  that .  jubft  dau  eargonei  m  lap  ror  htiiuid  dmftpti  m  ibr  dwanui 
locwoonweriw  fnpie»«kafiw  rvr>NT,p  i^erMf^mkily  Wk  knew  flir  iSiete  ■  * 

cnrfin  IphLkrr-lieTwren  ■ti'fLioi.e  .'■■•ne>  .ui:!  .aher  nrmninkiiTrialito.  Ir.T.nL.L 
«ri  frm-p«  in  lUTcreni  counties  are*  mrmauigjy  inlEitlepeakid  ut  a  w av  • — .  -,> 

i  jiulf  Litiit  y  dMi  iu:rr.; jd>h*  to  uudduee  fedhidL 

GiefiiUaiiofl  mil  .bewjip  nw^K of  ah« hd*r  Nttned  diK hplmie>  iLEJteiioai  in.  >;  umleniuixtL  tor  vj.nt* 
iTTirxlii.:mj;  >_  vm:r  d5  □u'Li.Triiw  iLjl  j  anil  llfcf  .  ‘.n'-tri  .imA  jlrrp.-,-r  Jum  Tbe  .vihudpjt  ^  jlryi.i:  ja.  nflMih;  anil 

oftkifi!  RAD.  «iil ineacJ  ^  toe  tiS  1  Jtv. r «il'>]anwmiLO!i  did Midiro.  dbiwr t«i» mom *t« tv.  ifllbraulin 
ajutf  the  t-Hhly  mrenlivTpl-nry  vavpim  ihouji*  ip  hei  itju*  Ttui  A  JWS  US-  NwihmI  (ietemii  Cnndl  iNRC! 
cucziulter  dLimeil  h-  FeinLlheai  unk  j  depliwiriLditj  rrtirrr.  when  .1  jitrmiuced  line  emubLird  rjiOjiiiry  offeiksal 
■LieiiLr  anil  »i’.  i—  1  >.uif  y,~  dn.  iurmi'  to  I  tr  -  Jim  iniiarrt  j!  tint  of  nuutdiis  ii  imr^iLtlk  mid  iruleaiinp  "  Mure 
Artioo.  li  QHdedlDMtall«f  aed  owkli  toac  iaflftdTMt  DM&tif  pettn.  A  m  nil  Ryiti  fhitt  Ce  NKL 

Cwrnuhoe  m  iNPitinJE  SMiw*  fuunri  hal  "dwTuadnecof  d#4  ndlRTifwi  nliedm  moddn  LOftrjpfrfnnrjtiii.f 

toil  JET  mu  >  ijd,  urjiTif  “1  ■•  .itH  Lir  »h -r  ill  i.j  Rik  L i  L-ilapr^ic  "  huidiu.  'll  «mliJ  be  Jkr  S5  ito  ,ar,  ini 

uid  if pt«hjf.  wicy  twli  iho  mcwo  dimity  n-jirne  ihe  ocodotra.'-  oufte  oflui  D  m  >mii  or'ihoo-nrm 

vul  uif^-liui!  bunt  uton  "  I 

Ur.  iiiii'  RAD  Il>  zuiaie.im!  Ji  Lis  bul  i  iwki,  tis  j  a  tail  i Tiler,  ha  nn  a  .  or  Wr  nenl  eenRinierr.. 

madris  tod  |ii>.'.  ctoizh  /rubies  ui  a  lufTieiEnl  ituaba  of  LUirilnn  10  pti'-tocr  nodsiafalr  uuulatjuu  of  Lir 

i-n.ii  of  4pfiiif«-  pfki/v  LtiOwfi  Toii  neod  t-aaH he  sai-.il nil  1>y  <  to*  phun  »  wodoJieft  bu  deeuadf die  anmuut 
«f  4  to  iki=  *.  In  jdy  ftmurkinits-  As  Jitwe  wunntiids  wwi  iwf  y  s;  ienhni  airn  mi  fhe«e  ams.  fly  ■HfeirvewM  of 
MRioepulbT  VUlLi  itfi**-.  ami  uf  t. aSLe  idLtvjii-  UP. 

jokD  H.HarWt*  H 

phn  M  Mlir-.jT[pr  ||i  u  elrkdof  nlltir  [>4k»  ni  Sclpnta  a"rP  Tacntp-'cip'Palkf  UpcatN*  cnrtra  nf-yr  Pi-il^Krt  of t™  Untto 
"rtitn.  kirtieh^lB".  Dt 

"ti  iFeiptu  |  kwth.S  iotiv*#-!  i  «ipirr  lc*«i5i  lOP.  i-UVleDOlj  limnpurTa«iHpn;l,^41Vwiiopnvtt^>«n<H>vnFin|Y>o 
U  S  rrv:™}-  LDd-psr  I  (  f  ««  O  A  C^.tPl-  PC  |b  OApid  tf 


wwnsdnnDHF^jii^  VlFh-ff  VLlI  ®fl  i'DOWYWDS 
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